Five essential data wrangling verbs

5.1. Subset columns with select()

5.1. Subset columns with select()

5.1. Subset columns with select()

flights.select(pl.col('flight','cost'))

5.1. Subset columns with select()

Often using the drop() method to remove columns is more efficient than select().

flights.drop(pl.col('cost'))

5.1. Subset columns with select()

• select() reduces the DataFrame to the essential columns and improves perfomance.

• Polars offers numerous options for more precise selections including:

  • Regular Expressions: flights.select(pl.col('^c.*$')) # columns starting with “c”
  • Select by data type: flights.select(df.select(pl.col(pl.Int64))
  • Slicing (not recommended, but possible!): flights[:]

5.2 Subset rows with filter()

The filter command identifies rows that you wish to include in a dataframe.

flights.filter(pl.col('non_stop'))

5.2 Subset rows with filter()

Let’s say that we wanted only the flights from RDU that were:

• a distance less than 1000 kilometers

flights.filter(pl.col('distance_km') < 1000)

5.2 Subset rows with filter()

Polars also has convenient tools for finding text:

• Is there a flight to Svalbard?

flights.filter(pl.col('flight').str.contains("Svalbard"))

5.2 Subset rows with filter()

Polars allows complex filter queries by combining multiple conditions using boolean expressions. For most operations, you will use the following operators to build these expressions.

5.2 Subset rows with filter()

flights.filter(
    (pl.col('distance_km')<6000) &
    (pl.col('cost')< 500.00)
)

5.3 sort()

sort() allows us to specify how the rows in the DataFrame should be ordered.

5.3 sort()

Let’s try a simple sort based on our filtered data.

• We will sort by distance for non-stop flights.

flights.filter(pl.col('non_stop')).sort(pl.col('distance_km'))

5.3 sort()

In the above, note how sort defaults to ascending order.

• If you want to sort in descending order:

flights.filter(pl.col('non_stop')).sort(pl.col('distance_km'), descending=True)

5.3 sort()

It is entirely possible to specify more than one column for a sort. - Let’s look at our non-stop flights sort by the cost and the distance.

(
flights
    .filter(pl.col('non_stop'))
    .sort(
        [
          pl.col('cost'),
          pl.col('distance_km')
        ],
        descending=[True, False]          
    )
)

5.3 sort()

If you have missing data in a sort column, it will be listed first.

• Show null values last by adding the nulls_last=True parameter:

flights.sort(pl.col('cost'), descending=True, nulls_last=True)

5.4 with_columns()

with_columns() provides a convenient way to add columns to DataFrames.

• First, we assign a column name using “keyword argument syntax” (cost_per_kilometer)
• Then we calculate the cost per kilometer

flights.with_columns(
   cost_per_kilometer = pl.col('cost') / pl.col('distance_km')
)

5.4 with_columns()

Polars only makes changes to dataframes permanent when you save the change.

• Even though I calculated “cost_per_kilometer” on the last slide…
• The column was not added to the DataFrame (any guesses as to why?)

flights

5.4 with_columns()

Polars only makes changes to dataframes permanent when you save the change.

• Let’s save the cost_per_kilometer permanently to our DataFrame.
• Note that this example assigns the output to flights:

flights = flights.with_columns(
   cost_per_kilometer = pl.col('cost') / pl.col('distance_km')
)
flights

5.4 with_columns()

Polars offers a second syntax style for creating columns.

• Expression based syntax assigns the column name using the .alias() method.

flights = flights.with_columns(
   (pl.col('cost') / pl.col('distance_km')).alias('cost_per_kilometer')
)
flights

5.5 group_by()

group_by() groups rows defined by one or more variables.

• Note that group_by() requires additional code to use the groupings!

flights.group_by(pl.col('non-stop')) # this creates groups... but no output!

<polars.dataframe.group_by.GroupBy at 0x10aa58830>

5.5 group_by()

How many flights are non-stop (true) and “multi-stop” (false)?

• group_by() combined with len() (length) will count the rows in each group!

flights.group_by(pl.col('non_stop')).len()

5.5 group_by()

If you have multiple aggregations that you wish to perform on each group of data

• agg()(or aggregrate) allows one or more calculations per grouping
• note: you must use expression based syntax if you use .agg() after a group_by()

(
flights.group_by(pl.col('non_stop'))
    .agg(
        pl.col('distance_km').mean().alias('average_distance_km')
    )
)

5.6 over()

I know…

I said I was only going to show you five common “verbs” in Polars…

• group_by() provides a convenient way to “collapse” data into groups…
• but, sometimes you want the group calculation in the original DataFrame
• This is where the over() method comes in handy.

5.6 over()

Let’s calculate the average distance over non-stop and multi-stop flights. - We will insert the calculation for each group in the DataFrame.

(
flights.
  with_columns(pl.col('distance_km')
    .mean()
    .over(pl.col('non_stop'))
    .alias('average_distance_km'))
  .select(pl.col('flight','non_stop','average_distance_km'))
)